Reducing Network Traffic and Managing Volatile Web Contents Using Migrating Crawlers with Table of Variable Information

As the size of the web continues to grow, searching it for useful information has become increasingly difficult. Also study reports that sufficient of current internet traffic and bandwidth consumption are due to the web crawlers that retrieve pages for indexing by the different search engines. Moreover, due to the dynamic nature of the web, it becomes very difficult for a search engine to provide fresh information to the user. An incremental crawler downloads modified contents only from the web for a search engine, thereby helps reducing the network load. This network load further can be reduced by using migrants. The migrants migrate to the web server for downloading, filtering and compressing the documents before transferring them to the search engine side. In this paper a more network efficient approach for extracting the volatile information from the web server using migrants with the help of table of volatile information has been developed, which further helps in reducing the network load significantly.